Information processing apparatus and information processing method

ABSTRACT

An information processing apparatus including circuitry that acquires information indicating a spatial relationship between a real object and a virtual object, and initiate generation of a user feedback based on the acquired information, the user feedback being displayed to be augmented to a generated image obtained based on capturing by an imaging device, or augmented to a perceived view of the real world, and wherein a characteristic of the user feedback is changed when the spatial relationship between the real object and the virtual object changes.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation application of U.S. patentapplication Ser. No. 16/566,477, filed on Sep. 10, 2019, which is acontinuation application of U.S. patent application Ser. No. 15/560,111,filed on Sep. 20, 2017, which is a National Phase Patent Application ofInternational Application No. PCT/JP2016/000871 filed on Feb. 18, 2016,and which claims priority from Japanese Patent Application JP2015-073561 filed on Mar. 31, 2015. Each of the above referencedapplications is hereby incorporated by reference in its entirety.

TECHNICAL FIELD

The technology disclosed in present disclosure relates to an informationprocessing apparatus, an information processing method, and a computerprogram which processes an Augmented Reality (AR) object displayed in areal space observed by a person.

BACKGROUND ART

AR technology is known which enhances the real world observed by aperson, by adding visual information such as a virtual object in a realspace. According to AR technology, a user can be made to perceive avirtual object (hereinafter, called an “AR object”) so as if it ispresent in a real space. A head mounted display, used by a personwearing it on his or her head, a small-sized information terminal suchas a head-up display, a smartphone or a tablet, a navigation system, agame device or the like can be included as a display apparatus whichmakes a user visually recognize an AR object at the same time as animage of a real space. By controlling a binocular parallax, aconvergence of both eyes, and a focal length in these display apparatus,an AR object can be made to be stereoscopically viewed. Further, byperforming a control which changes the drawing of an AR objectcorresponding to a shadow, a viewpoint position, or a change in a visualline direction, a stereoscopic feeling of the AR object can be produced.

A dialogue system can also be considered in which a person performs anoperation to an AR object by a hand or a finger. However, since an ARobject is a virtual object not actually present, a sense of touch is notobtained, even if a person performs a contacting or pressing operation,and so there will be a problem such as an operation by a user beingdifficult to understand.

For example, an information processing apparatus has been proposed whichperforms feedback of an operation by stereoscopically displaying aparticle, when detecting that a hand of a user has entered into a spaceregion detected by an operation on the space (for example, refer to PTL1). According to such an information processing apparatus, a user canvisually recognize that his or her hand has entered into a space regioncapable of detecting an operation. However, since visual feedback suchas a display of a particle is not able to be given at the time when notentering into a space region capable of detecting, it will be difficultto obtain a specific position relationship or depth information such aswhether the hand of the user himself or herself is in front or behindthe space region, or whether the hand of the user himself or herself isclose to or far from the space region.

CITATION LIST Patent Literature

[PTL 1]

JP 2012-256104A SUMMARY Technical Problem

The present inventors of the technology disclosed in the presentdisclosure have provided an excellent information processing apparatus,information processing method, and computer program capable of suitablyprocessing a virtual object visually recognized by a user at the sametime as an image of a real space.

Solution to Problem

According to an embodiment of the present disclosure, there is providedan information processing apparatus including circuitry configured toacquire information indicating a spatial relationship between a realobject and a virtual object, and initiate generation of a user feedbackbased on the acquired information, the user feedback being displayed tobe augmented to a generated image obtained based on capturing by animaging device, or augmented to a perceived view of the real world,wherein a characteristic of the user feedback is changed when thespatial relationship between the real object and the virtual objectchanges.

Further, according to an embodiment of the present disclosure, there isprovided an information processing method including acquiringinformation indicating a spatial relationship between a real object anda virtual object, generating a user feedback based on the acquiredinformation and displaying the user feedback to be augmented to agenerated image obtained based on capturing by an imaging device, oraugmented to a perceived view of the real word, wherein a characteristicof the user feedback is changed when the spatial relationship betweenthe real object and the virtual object changes.

Further, according to an embodiment of the present disclosure, there isprovided a non-transitory computer-readable medium having embodiedthereon a program, which when executed by a computer causes the computerto execute a method including acquiring information indicating a spatialrelationship between a real object and a virtual object, generating auser feedback based on the acquired information; and displaying the userfeedback to be augmented to a generated image obtained based oncapturing by an imaging device, or augmented to a perceived view of thereal word, wherein a characteristic of the user feedback is changed whenthe spatial relationship between the real object and the virtual objectchanges.

Advantageous Effects of Invention

According to one or more embodiments of the technology disclosed in thepresent disclosure, an excellent information processing apparatus,information processing method, and computer program can be provided,which can add a visual effect showing an operation by a real object to avirtual object.

Note that, the effect described in the present disclosure is merely anexample, and the effect of the present disclosure is not limited tothis. Further, the present disclosure will often accomplish furtheradditional effects, other than the above described effect.

It is further desirable for the features and advantages of thetechnology disclosed in the present disclosure to be clarified by a moredetailed description based on the attached embodiments and figures,which will be described below.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a figure which shows a state in which a user wearing atransmission-type head mounted display 100 is viewed from the front.

FIG. 2 is a figure which shows a state in which a user wearing the headmounted display 100 is viewed from above.

FIG. 3 is a figure which shows a state in which a user wearing animmersive-type head mounted display 300 is viewed from the front.

FIG. 4 is a figure which shows a state in which a user wearing the headmounted display 300 is viewed from above.

FIG. 5 is a figure which schematically shows an internal configurationexample of the head mounted display 100 shown in FIG. 1 and FIG. 2.

FIG. 6 is a figure which shows an example of a method for understandinga position relationship between an AR object and a real object.

FIG. 7 is a figure which shows another example of a method forunderstanding a position relationship between an AR object and a realobject.

FIG. 8 is a figure which shows a state in which a shadow of a hand of auser is drawn on the surface of an AR object.

FIG. 9 is a figure which illustrates visual feedback corresponding to adistance between a real object and an AR object.

FIG. 10 is a figure which illustrates visual feedback corresponding to adistance between a real object and an AR object.

FIG. 11 is a figure which illustrates visual feedback corresponding to adistance between a real object and an AR object.

FIG. 12 is a figure which illustrates visual feedback corresponding to adistance between a real object and an AR object.

FIG. 13 is a figure which illustrates visual feedback corresponding to adistance between a real object and an AR object.

FIG. 14 is a figure which shows an effective range capable of operatingan AR object.

FIG. 15 is a figure which illustrates visual feedback corresponding to adistance between a real object and an AR object.

FIG. 16 is a figure which illustrates visual feedback corresponding to adistance between a real object and an AR object.

FIG. 17 is a figure which illustrates visual feedback corresponding to adistance between a real object and an AR object.

FIG. 18 is a figure which illustrates visual feedback corresponding to adistance between a real object and an opposite surface side of an ARobject.

FIG. 19 is a figure which illustrates visual feedback corresponding to adistance between a real object and an AR object.

FIG. 20 is a figure which illustrates visual feedback corresponding to adistance between a real object and an AR object.

FIG. 21 is a figure which shows an example in which an AR object isdrawn with a contrasting color to that of visual feedback.

FIG. 22 is a figure which shows an example in which visual feedback isdrawn with a contrasting color to that of an AR object.

FIG. 23 is a figure which shows an example in which an effective rangeof visual feedback is limited.

FIG. 24 is a figure which schematically shows a functional configurationfor performing visual feedback for an operation of a user to an ARobject.

FIG. 25 is a flow chart which shows a process procedure for performing adrawing process for an AR image attached to visual feedback.

FIG. 26 is a flow chart which shows a process procedure for drawingvisual feedback of a real object approaching an AR object.

FIG. 27 is a flow chart which shows a process procedure for drawingvisual feedback of a real object behind an AR object.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the technology disclosed in the presentdisclosure will be described in detail while referring to the figures.

A. System Configuration

FIG. 1 shows a state in which a user wearing a transmission-type(see-through) head mounted display 100 is viewed from the front, as anexample of a device which presents visual information including an ARobject. The user wearing the transmission-type head mounted display 100can observe the surroundings (real world) through a display image.Therefore, the head mounted display 100 can cause a virtual displayimage such as an AR object to be viewed overlapping the scenery of thereal world.

The head mounted display 100 shown in FIG. 1 is constituted from astructure similar to that of glasses for vision correction. The headmounted display 100 has transparent virtual image optical units 101L and101R respectively arranged at positions facing the left and right eyesof the user, and has an enlarged virtual image of an image observed bythe user (an AR object or the like) formed. Each of the virtual imageoptical units 101L and 101R are supported by a glasses frame-typesupporting body 102.

Further, microphones 103L and 103R are arranged in the vicinity of boththe left and right ends of the supporting body 102. By approximatelyleft-right symmetrically including the microphones 103L and 103R at thefront surface, and by recognizing only audio located at the center (thevoice of the user), noise of the surroundings and other people's voicescan be separated, and an incorrect operation can be prevented, forexample, at the time of an operation by audio input.

FIG. 2 shows a state in which the head of the user wearing the headmounted display 100 shown in FIG. 1 is viewed from above.

As illustrated, display panels 104L and 104R, which respectively displayand output images for the left eye and the right eye, are arranged atboth the left and right ends of the head mounted display 100. Each ofthe display panels 104L and 104R are constituted from a micro displaysuch as a liquid crystal display or an organic EL element (©LED: OrganicLight-Emitting Diode). The display panels 104L and 104R can display anAR object or the like overlapping on the scenery of the surroundings(the real word) observed by the user. Left and right display imagesoutput from the display panels 104L and 104R are guided up until thevicinity of each of the left and right eyes by the virtual image opticalunits 101L and 101R, and these enlarged virtual images are focused onthe eyes of the user. While a detailed illustration is omitted, thevirtual image optical units 101L and 101R each include an optical systemwhich collects irradiation light from the micro display, a light guideplate arranged at a position where passing light of the optical systemis incident, a deflection filter which reflects incident light to thelight guide plate, and a deflection filter which causes light spread bytotal reflection within the light guide plate to be emitted towards theeye of the user.

Note that, while an illustration is omitted in FIG. 1 and FIG. 2, thehead mounted display 100 may additionally include an outside camerawhich photographs the scenery in a visual line direction of the user. Byapplying a process such as image recognition to a photographic image ofthe outside camera, a real object (for example, a hand of the user, apointer operated by the user or the like) which performs an operation toan AR object (or this enlarged virtual image) displayed on the displaypanels 104L and 104R can be specified, and this position and posture canbe measured.

Further, FIG. 3 shows a state in which a user wearing an immersive-typehead mounted display 300 is viewed from the front, as an example of adevice which presents visual information including an AR object.

The immersive-type head mounted display 300 directly covers the eyes ofthe user at the time when worn by the user on his or her head or face,and gives a sense of immersion to the user while viewing an image.Further, different to the transmission-type head mounted display 100,the user wearing the immersive-type head mounted display 300 is not ableto directly view the scenery of the real world. However, by displaying acaptured image of an outside camera 305, which photographs the sceneryin a visual line direction of the user, the user can indirectly view thescenery of the real world (that is, observe the scenery by a videosee-through). It is needless to say that a virtual display image such asan AR image can be viewed overlapping with such a video see-throughimage.

The head mounted display 300 shown in FIG. 3 has a structure resemblinga hat shape, and is constituted so as to directly cover the left andright eyes of the user who is wearing it. Display panels 304L and 304Rwith which the user observes are respectively arranged at positionsfacing the left and right eyes on the inside of the main body of thehead mounted display 300. The display panels 304L and 304R areconstituted, for example, by a micro display such as an organic ELelement or a liquid crystal display. A captured image of the outsidecamera 305 can be displayed as a video see-through image on the displaypanels 304L and 304R, and an AR object can be additionally overlapped onthis video see-through image.

The outside camera 305 for a surrounding image (visual field of theuser) input is provided in approximately the center of the main bodyfront surface of the head mounted display 300. The outside camera 305can photograph the scenery in a visual line direction of the user.Further, by applying a process such as image recognition to the outsidecamera 305, a real object (for example, a hand of the user, a pointeroperated by the user or the like) can be specified, and this positionand posture can be measured.

Further, microphones 303L and 303R are respectively provided in thevicinity of both the left and right ends of the main body of the headmounted display 300. By approximately left-right symmetrically holdingthe microphones 303L and 303R, and by recognizing only audio located atthe center (tile voice of the user), noise of the surroundings and otherpeople's voices can be separated, and an incorrect operation can beprevented, for example, at the time of an operation by audio input.

FIG. 4 shows a state in which the head of the user wearing the headmounted display 300 shown in FIG. 3 is viewed from above.

The illustrated head mounted display 300 holds the display panels 304Land 304R for the left eye and the right eye on the side facing the faceof the user. The display panels 304L and 304R are constituted, forexample, by a micro display such as an organic EL element or a liquidcrystal display. Display images of the display panels 304L and 304R areobserved by the user as enlarged virtual images by passing through thevirtual image optical units 301L and 301R. Further, since there will bepersonal differences for each user for the eye height and theinterpupillary distance, it may be necessary for each of the left andright display systems to perform position alignment with the eyes of theuser who is wearing them. In the example shown in FIG. 4, aninterpupillary adjustment mechanism 306 is included between the displaypanel for the right eye and the display panel for the left eye.

FIG. 5 schematically shows an internal configuration example of the headmounted display 100 shown in FIG. 1 and FIG. 2. However, for the sake ofconvenience, different reference numerals will be attached in FIG. 5,even if the parts are the same as those of FIG. 1 and FIG. 2. Further,the internal configuration of the head mounted display 300 shown in FIG.3 and FIG. 4 may also be understood as being the same as that of FIG. 5.Hereinafter, each of the units will be described.

A control unit 501 includes a Read Only Memory (ROM) 501A and a RandomAccess Memory (RAM) 501B. Program codes executed by the control unit 501and various types of data are stored within the ROM 501A. The controlunit 501 starts a display control of an image, by executing a programloaded in the RAM 501B, and integrally controls all of the operations ofthe head mounted display 100. Navigation and games, and also variousapplication programs which render an AR image visually recognized by auser at the same time as an image of a real space, can be included asprograms stored in the ROM 501A and executed by the control unit 501.Further, in the control unit 501, a display process is performed for aphotographic image of an outside camera 504 (or an environment camera703, which will be described below), and a photographic subject or areal object is specified by performing image recognition for aphotographic image as necessary. However, other than being executedwithin the head mounted display 100 (display apparatus main body) whichdisplays an AR image, a process which renders an AR image (which will bedescribed below) can be configured so as to be executed by an externalapparatus such as a server on a network, or to execute only a displayoutput by the head mounted display 100 by receiving this calculationresult by a communication unit 505.

An input operation unit 502 includes one or more operators for a user toperform an input operation, such as keys, buttons, or switches, acceptsan instruction of the user via the operators, and outputs the acceptedinstruction to the control unit 501. Further, the input operation unit502 accepts an instruction of the user constituted from a remote controlcommand received from a remote control (not illustrated) by a remotecontrol reception unit 503, and outputs the accepted instruction to thecontrol unit 501.

An outside camera 504 is arranged at approximately the center of themain body front surface of the head mounted display 100, for example,and photographs the scenery in a visual line direction of the user, forexample. The outside camera 504 may include a rotation movement functionor a viewing angle change (zoom) function in each direction of a pan,tilt, and roll. The user may instruct a posture of the outside camera504, through the input operation unit 502.

The communication unit 505 performs a communication process with anexternal device, and a modulation-demodulation and encoding-decodingprocess of a communication signal. A content reproduction apparatuswhich supplies viewing content (a Blu-ray Disc or DVD player), amultifunctional information terminal such as a smartphone or a tablet, agame device, a streaming server or the like can be included as acommunicating external apparatus. Further, the control unit 501 sendstransmission data to the external apparatus from the communication unit505.

The configuration of the communication unit 505 is arbitrary. Forexample, the communication unit 505 can be configured, in accordancewith a communication system used for a transmission and receptionoperation with an external apparatus which becomes a communicationpartner. The communication system may be any wired or wireless form. AMobile High-definition Link (MHL), a Universal Serial Bus (USB), a HighDefinition Multimedia Interface (HDMI) (registered trademark), Wi-Fi(registered trademark), Bluetooth (registered trademark) communication,Bluetooth (registered trademark) Low Energy (BLE) communication,ultra-low power consumption wireless communication such as ANT,IEEE802.11s or the like can be included as the communication systemstated here. Alternatively, the communication unit 505 may be a cellularwireless transmission and reception device, for example, which operatesin accordance with standard specifications such as Wideband CodeDivision Multiple Access (W-CDMA) or Long Term Evolution (LTE).

The storage unit 506 is a large capacity storage apparatus constitutedby a Solid State Drive (SSD) or the like. The storage unit 506 storesapplication programs executed by the control unit 501 and various typesof data. Further, moving images or still images photographed by theoutside camera 505 may be stored within the storage unit 506.

The image processing unit 507 additionally performs a signal processsuch as image quality correction to an image signal output from thecontrol unit 501, and performs conversion into a resolution matched withthe screen of the display unit 509. Also, a display driving unit 508supplies a pixel signal based on a signal-processed image signal, bysequentially selecting and line sequentially scanning pixels of thedisplay unit 509 for each row.

The display unit 509 has a display panel constituted by a micro displaysuch as an organic EL element or a liquid display panel, for example. Avirtual image optical unit 510 performs an enlargement projection for animage such as an AR object displayed on the display unit 509, and causesthe enlargement-projected image to be observed by the user as anenlarged virtual image. As a result of this, the user can visuallyrecognize an AR object at the same time as an image of a real space.

An audio processing unit 511 additionally performs sound qualitycorrection, audio amplification, or signal processing of an input audiosignal or the like, to an audio signal output from the control unit 501.Also, an audio input and output unit 512 performs external output forthe audio after audio processing, and audio input from a microphone(described above),

AR technology is already widely used. According to AR technology, a usercan be made to perceive a virtual object (hereinafter, called an “ARobject”) so as if it is present in a real space. Further, by controllinga binocular parallax, a convergence of both eyes, and a focal length, anAR object can be made to be stereoscopically viewed. Further, byperforming a control which changes the drawing of an AR objectcorresponding to a shadow, a viewpoint position, or a change in a visualline direction, a stereoscopic feeling of the AR object can be produced.In addition, a dialogue system can also be considered in which a personperforms an operation to an AR object by a hand or a finger. However,since an AR object is a virtual object not actually present, a sense oftouch is not obtained, even if a person performs a contacting orpressing operation, and so it will be difficult for an operation to beunderstood.

Accordingly, image display technology is proposed, in which is easy tointuitively operate an AR object even if a sense of touch is notobtained by contacting or pressing, by presenting visual feedback to anAR object based on a position relationship with a real object (forexample, a hand of a user attempting to operate the AR object), as thetechnology disclosed in embodiments of the present disclosure. Alocation of providing feedback, according to embodiments, may be basedon a location of a target whose position is indicated by a trajectorydirection of the real object, but is not limited thereto.

Here, a method for understanding a position relationship between an ARobject and a real object will be described.

FIG. 6 shows an example of a method for understanding a positionrelationship between an AR object and a real object. The same figureshows a state in which a user wearing the head mounted display 100 isattempting to operate an AR object 601 displayed by the head mounteddisplay 100 with a hand 602 of the user himself or herself. Here, thehand 602 of the user is made a measurement target.

The AR object 601 holds a prescribed shape and size. In the exampleshown in FIG. 6, in order for simplification, the AR object 601 isarranged in an approximately horizontal plane parallel with the front ofthe face of the user. Further, the AR object 601 holds a position andposture provided in a real space. The head mounted display 100 rendersthe AR object 601, so as to be arranged at this position and posture,displays the rendered AR object 601 on the display unit 509, andperforms observation to the user through the virtual image optical unit510.

The outside camera 504 is arranged at approximately the center of themain body front surface of the head mounted display 100, and photographsthe scenery in a visual line direction of the user. When the hand 602 ofthe user enters into a photographic range 603 of the outside camera 504,a position in the real space of the hand 602 of the user within aphotographic image can be measured, through a process such as imagerecognition.

In order to easily set depth information of the hand 602 of the user, astereoscopic camera may be applied to the outside camera 504, or adistance sensor may be used as well. Further, detection may be easilyset from a photographic image of the outside camera 504, by attachingone or a plurality of markers (not illustrated) to a real object whichbecomes a measurement target, such as the hand 602 of the user.

Note that, strictly speaking, a display coordinate system of the displayunit 509 which displays an AR object (or a projection coordinate systemwhich projects an enlarged virtual image of a display image), and aphotographic coordinate system of the outside camera 504 whichphotographs a real object which becomes a measurement target, do notcompletely match. Hereinafter, in order for a simplification of thedescription, the display coordinate system and the photographiccoordinate system matching or having an error difference will bedisregarded, or the succeeding processes will be performed by performinga conversion into an absolute coordinate system.

Further, FIG. 7 shows another example of a method for understanding aposition relationship between an AR object and a real object. The samefigure shows a state in which a user wearing the head mounted display100 is attempting to operate an AR object 701 displayed by the headmounted display 100 with a hand of the user himself or herself 702.Here, the hand of the user 702 is made a measurement target (same asabove).

The AR object 701 holds a prescribed shape and size. In the exampleshown in FIG. 7, in order for simplification, the AR object 701 isarranged in an approximately horizontal plane parallel with the front ofthe face of the user. Further, the AR object 701 holds a position andposture provided in a real space. The head mounted display 100 rendersthe AR object 701, so as to be arranged at this position and posture,displays the rendered AR object 701 on the display unit 509, andperforms observation to the user through the virtual image optical unit510.

An environment camera 703 is provided on the ceiling or a wall of a roomin which the user is present, and performs photography so as to lookdown on a real space (or a working space of the user) in which the ARobject 701 is overlapping. When the hand of the user 702 enters into aphotographic range 704 of the environment camera 703, a position in thereal space of the hand of the user 702 within a photographic image ismeasured, through a process such as image recognition.

Note that, the environment camera 703 may be supported by a platform(not illustrated) rotationally moving in each direction of a pan, tilt,and roll. Further, while only one environment camera 703 is drawn inFIG. 7 in order for simplification, two or more environment cameras maybe used, in order to obtain three-dimensional position information ofthe hand of the user 702 which is a measurement target, or in order toenlarge the photographic range 704 (or to not have blind spotsoccurring).

However, in the context of implementing the technology disclosed inembodiments of the present disclosure, the method which obtains positioninformation of a real object such as a hand of a user is not limited tothe methods shown in FIG. 6 and FIG. 7.

B. Visual Feedback Example Corresponding to a Position RelationshipBetween an AR Object and a Real Object (1)

An AR object, which the head mounted display 100 displays overlapping ona real space, retains position information which includes depthinformation from a user wearing the head mounted display 100. Also, whenmeasuring position information of a real object (a hand of the user orthe like) which is attempting to perform an operation to an AR object,by the methods shown in FIG. 6 or FIG. 7, or a method other than these,depth information between the AR object and the real object is compared,it is determined whether the real object is in front of the AR object,is touching the AR object, or is behind the AR object, and visualfeedback is given to the user based on this determination result.

In the case where the real object is in front of the AR object, a usercan be made to recognize the AR object as at a position far from thehand of the user himself or herself, by performing a hidden surfaceprocess so that the AR object becomes invisible by being hidden by thereal object (set so that a part or the entire AR object is not drawn).

FIG. 8 shows a state in which a shadow of a hand of a user is drawn onthe surface of an AR object, as an example of visual feedback. In theillustrated example, a virtual light source 801 is provided in thevicinity of a visual line of a user (or the outside camera 504), andirradiation light 802 of this virtual light source 801 draws a shadow804 shielded by the hand of the user 803 on the surface of an AR object805. Since a shadow gives a sense of presence or reality to AR, andexpresses the presence of a shielding object, it will be a great help tothe spacial understanding of the user.

When the virtual light source 801 is made a point light source, thisirradiation light 802 extends in a spherical shape. Therefore, theshadow of the hand 804 will become smaller as the hand 803 becomescloser to the AR object 805, and will have a clear outline. Further,when the hand 803 touches the AR object 805, the shadow will almostbecome invisible. Conversely, the shadow of the hand 804 will becomelarger as the hand 803 becomes distant from the AR object 805, and willhave a blurred outline.

The head mounted display 100 may present feedback by senses other thanvisually, such as making a warning sound louder in accordance with adistance between the AR object and the real object (or conversely,making quieter), making an amplitude or frequency of vibrations added tothe head of the user larger (or conversely, making smaller), or applyingheat, in parallel with visual feedback such as the above describeddrawing a shadow of the real object on the surface of the AR object.Accordingly, the head mounted display 100 may additionally include adevice which presents feedback by senses, such as a piezo actuator forgenerating vibrations, or a heat generation device. By using incombination one or more types of feedback which are presented by sensesother than the visual field, with visual feedback using a shadow of areal object, the spacial understanding of an AR object by a user can beadditionally supported.

FIG. 9 to FIG. 13 illustrate states in which visual feedback presentedto a user is changed in stages in accordance with a distance between areal object and an AR object. However, a position relationship betweenan AR object arranged in a real space, and a finger of a user attemptingto operate the AR object, is shown on the right side of each figure, anda display example of an AR object at the time of each positionrelationship (an image observed by a user) is shown on the left side ofeach figure. However, since the real object actually exists and the ARobject is virtually present, the “distance” between the real object andthe AR object used here is a virtual distance. This virtual distance iscalculated based on information of a position at which the AR object isarranged in a virtual real space, and information of a position of adetected real object. Further, the AR object can include a display usedfor an input operation performed by a user for a real space or a virtualspace. A selection operation of options, a pointing operation (inputoperation of coordinates), or a text input can be included as an inputoperation of a user. In the examples shown in FIG. 9 to FIG. 13, threemenu buttons are included as Graphical Use Interface (GUI) parts.

In the example shown in FIG. 9, a hand of a user 901 is separated farfrom an AR object 902. Therefore, a shadow of the hand 903 is large, andthis outline is drawn blurred.

In the example shown in FIG. 10, a hand 1001 is approaching an AR object1002. Therefore, whether the hand 1001 is approaching the vicinity ofthe AR object 1002 is visually expressed, by making a shadow of the hand1003 smaller than that of the example shown in FIG. 9, and drawing thisoutline sharper.

In the example shown in FIG. 11, a hand 1101 is touching an AR object1102. Therefore, a shadow of the hand is almost invisible. Further,whether the vicinity of the AR object 1102 appears to be touched by thehand 1101 (the location which becomes a shortest distance from the tipof the hand 1101) can be calculated, based on a comparison result ofdepth information between the hand 1101 and the AR object 1102. Asillustrated, the spacial understanding of the user can be additionallyhelped, by performing a drawing process such as causing the portion atwhich the hand 1101 appears to be touching to be lit up. In the exampleshown in FIG. 11, a menu button 1103 touched by the hand 1101 isdisplayed highlighted.

In the example shown in FIG. 12, the tip of a hand 1201 is additionallypushed forward after touching an AR object 1202, and is intersectingwith the AR object 1202. The portion, of the tip of the hand 1201,pushed through from the AR object 1202 can be calculated, based on acomparison result of depth information between the hand 1201 and the ARobject 1202. Then, when the depth at which the hand 1201 pushes throughthe AR object 1202 becomes a fixed depth or more, it will be expressedso that the pushed through portion is hidden, by displaying the ARobject 1202 overlapping on this portion. Further, by drawing an alarm(warning feedback) such as causing the portion of the AR object 1202pushed through by the hand 1202 to be lit up, a warning can be performedso as to finish without additional movement by the user.

In the example shown in FIG. 13, when a hand 1301 is additionally pushedthrough from an AR object 1302, not only is it no longer possible tooperate the AR object 1302, but there is a risk that the hand 1301 willbe injured, by colliding with an obstacle (another real object) such asa wall 1303 behind the AR object 1302. In such a case, the user can bemade to visually recognize an object which is behind, by causing thedisplay of the AR object 1302 to disappear (or causing it to flashon-off), or by displaying warning feedback such as making itsemitransparent. Further, at the time when a real object enters into aprescribed warning range, such as at the time when appearing to hit awall which is behind or the like, the risk may be avoided, by usingtogether with feedback for senses other than visually, such as a warningsound or a vibrator. While an illustration is omitted, warning feedbackmay be performed such as changing the color or shape of another realobject appearing to collide such as the wall 1303, and the risk may besurely notified to a user.

As shown in FIG. 11, it becomes possible to operate an AR object, in astate where a hand has touched the AR object. However, since there is nofeedback by a sense of touch even if touching an AR object, it will be adifficult task to cause the hand to move in a depth direction justtouching the AR object, or to keep the hand at this depth position.Accordingly, as shown in FIG. 14, an effective range of operation 1403by a real object may be set, within a prescribed distance close to infront and behind the depth direction from an AR object 1402. When a handof the user 1401 enters into the effective range of operation 1403, byregarding that it is touching the AR object 1402, and executing orcontinuing execution of an operation to the AR object 1402 for which thehand 1401 has moved somewhat in a forward or backward direction, thesuccess rate of the operation by the hand of the user can be improved.

Note that, the user may be prompted to correct a position of the hand1401 (real object), by being notified that there is actually no contactwith the AR object 1402, while allowing a continuous operation of the ARobject 1402 within the effective range of operation 1403.

C. Visual Feedback Example Corresponding to a Position RelationshipBetween an AR Object and a Real Object (2)

In the examples shown in FIG. 8 to FIG. 10, a shadow was used for visualfeedback of a distance between a real object and an AR object. Since ashadow gives a sense of presence or reality to AR, and expresses thepresence of a shielding object, it will be a great help to the spacialunderstanding of a user. However, the calculation burden for generatingand drawing a shadow in real-time is high, and it will be difficult todraw a complete shadow. In the case where the surface of an AR objecthas a shape which is uneven and not smooth, the drawing of a shadow willbecome additionally difficult.

Accordingly, instead of a method which draws a shadow of a real objecton an AR object, a presentation method of visual feedback is proposedwhich draws a ring-shaped indicator of light corresponding to a distancebetween a real object and an AR object.

FIG. 15 to FIG. 17 illustrate states in which a ring-shaped indicator oflight changes in stages in accordance with a distance between a realobject and an AR object. However, a position relationship between an ARobject arranged in a real space, and a finger of a user attempting tooperate the AR object, is shown on the right side of each figure, and adisplay example of an AR object at the time of each positionrelationship (an image observed by a user) is shown on the left side ofeach figure. In order for simplification, a description will be made byusing a smooth AR object constituted by a plain texture.

In the example shown in FIG. 15, a hand 1501 is separated far from an ARobject 1502. Therefore, a ring-shaped indicator of light 1503constituted by a large and blurred line is drawn, centered on a pointwhich becomes a shortest distance to the hand 1501, on the surface ofthe AR object 1502.

In the example shown in FIG. 16, a hand 1601 is approaching an AR object1602. Therefore, a ring-shaped indicator of light 1603 constituted by asmaller and sharper line than that shown in FIG. 15 is drawn, centeredon a point which becomes a shortest distance to the hand 1601, on thesurface of the AR object 1602.

In the example shown in FIG. 17, a hand 1701 is touching an AR object1702 (or enters into an effective range (described above)). A small andsharp ring-shaped indicator of light 1703 is drawn, which specifies thelocation at which the hand 1701 has touched the AR object 1702.

D. Visual Feedback Example Corresponding to a Position RelationshipBetween an AR Object and a Real Object (3)

FIG. 8 to FIG. 17 are basically examples of visual feedback performed inthe case where operating an AR object from the front surface. In thecase where an AR object is arranged separated to some extent from auser, or in the case where an AR object is provided on a desk surface orthe like, it may be necessary for the user to operate the AR object fromthe front surface. Therefore, as can be understood from FIG. 9 to FIG.11, for example, in the case where operating an AR object from the frontsurface, at least a part of the surface of the AR object will be hiddenby the hand of the user and will be difficult to see, and the AR objectwill additionally become difficult to see when casting a shadow asvisual feedback.

On the other hand, in the case where an AR object can be arranged at aposition close to the user, the hand will reach to the opposite surfaceside of the AR object. By operating an AR object from the oppositesurface, it is possible to perform an operation without shielding the ARobject.

FIG. 18 to FIG. 20 illustrate states in which an AR object the same asthat of the examples shown in FIG. 9 to FIG. 13 is operated by a userfrom an opposite surface side. However, a position relationship betweenan AR object arranged in a real space, and a finger of a user attemptingto operate the AR object, is shown on the right side of each figure, anda display example of an AR object at the time of each positionrelationship (an image observed by a user) is shown on the left side ofeach figure. The AR object is a GUI of an application or game beingexecuted, for example, and includes three menu buttons.

In the example shown in FIG. 18, a hand of a user 1801 is separated fromthe opposite surface of an AR object 1802. At this stage, visualfeedback is not performed at all for the AR object 1802. This is becausethe AR object 1802 is not able to be operated from the separated hand ofthe user 1801.

In the example shown in FIG. 19, a hand of a user 1901 is approachingthe opposite surface of the AR object 1902. Therefore, visual feedbackof the hand 1901 approaching is performed, by emphasizing a display of amenu button 1903, which corresponds to the location at which the hand ofthe user 1901 is the closest, more than the surroundings.

In the example shown in FIG. 20, a hand of a user 2001 is in contactwith the opposite surface of an AR object 2002. Therefore, visualfeedback of being touched by the hand 2001 is performed, by switching toa display state in which a menu button 2003 touched by the hand of theuser 2001 from the opposite surface side is selected. Note that, it maybe a state in which the menu button 2003 is selected, not only the handof the user 2001 being completely in contact with the opposite surfaceof the AR object 2002, but also by entering into an effective rangewithin a fixed distance from the opposite surface of the AR object 2002such as shown in FIG. 14.

If it is an operation from the opposite surface side of an AR objectsuch as shown in FIG. 18 to FIG. 20, the operation can be performed,without shielding the AR object by a real object such as the hand of auser.

E. Color Adjustment of Visual Feedback

FIG. 8 to FIG. 10 have illustrated visual feedback such as drawing ashadow, of a real object in front of an AR object, on the surface of theAR object corresponding to a distance from the AR object. In this way,when visual feedback is drawn on the surface of an AR object, it ispossible to cause the visibility of the AR object to be reduced.Accordingly, in the case where visual feedback is drawn on the surfaceof an AR object, the color may be enhanced, and visual feedback may beperformed while retaining the visibility of the user, by mutuallydrawing the color of the AR object and the color of the visual feedbackas contrasting colors.

FIG. 21 shows a state in which a color of a region overlapping with ashadow of the hand of a user 2102, which is visual feedback from withinthe AR object 2101, is drawn with a contrasting color to that of theshadow of the hand 2102. Further, FIG. 22 conversely shows a state inwhich a color of a region overlapping with an AR object 2201 from withina shadow of the hand of a user 2202, which is visual feedback, is drawnwith a contrasting color to that of the AR object 2201. In both of theexamples of FIG. 21 and FIG. 22, a shadow of the forefinger of the useris overlapping with one character “N” from within a character string of“Today's News” drawn on the AR object. As described above, by mutuallydrawing the color of the AR object and the color of visual feedback ascontrasting colors, the character “N” can be recognized, without beingburied in the shadow of the forefinger.

F. Limitation of the Feedback Range

FIG. 6 showed a method for understanding a position of a real object (ahand of a user or the like) by using the outside camera 504 included onthe head mounted display 100. Further, FIG. 7 showed a method forunderstanding a position of a real object by using the environmentcamera 703 provided on the ceiling or a wall of a room in which a useris present. According to these methods, since a position relationshipbetween a real object and an AR object can be understood, if in aphotographic range of the camera, it is possible to perform visualfeedback in the entire photographic range.

However, in the case where a hand of a user is made a target of visualfeedback, it will be sufficient if a position relationship can beunderstood at the time when overlapping with an AR object, and it maynot be necessary to perform visual feedback in the entire photographicrange of the camera.

FIG. 23 shows an example in which an effective range of visual feedbackis limited. In the same figure, an effective range of visual feedback islimited to a fixed region 2302 enclosing an AR object 2301 viewed by auser. Further, an effective range of visual feedback is limited to afixed section 2303 including the AR object 2301 viewed by a user, in adepth direction.

By limiting such an effective range of visual feedback, wastefulnesssuch as performing useless visual feedback can be prevented, for anoperation in which it is clear that there is not operation to an ARobject, such as a user moving a hand around his or her face, forexample.

That is, by limiting an effective range of visual feedback, the drawingload can be reduced, and an operation of a user being obstructed due touseless visual feedback can be prevented.

G. Functional Configuration

FIG. 24 schematically shows a functional configuration for performingvisual feedback for an operation of a user to an AR object. As describedabove, visual feedback is basically implemented by giving a visualeffect to an image of an AR object, in accordance with positionrelationship information showing a position relationship (distance of adepth direction or the like) between the AR object, and a real objectperforming an operation instruction to the AR object. While acalculation process for performing visual feedback can be executedwithin a display apparatus main body which displays an AR object such asthe head mounted display 100, a calculation process may be performed forperforming visual feedback in an external apparatus such as a server ona network, or only a display output may be performed in accordance withthis calculation result in the display apparatus. In the case of thelatter, the display apparatus may transmit a result which measuresposition information of a real object (a hand of a user or the like) tothe server. Further, it is also possible for a calculation process forperforming visual feedback to be implemented, not as a special hardwareapparatus, but by a computer program written in a computer-readableformat so as to implement prescribed processes on a computer. Such acomputer program is recorded in a computer-readable storage medium, oris delivered via a wired or wireless communication route.

The AR information acquisition unit 2401 acquires AR information such asa position, posture, size, or texture related to an AR object to bedisplayed overlapping on a real space, generated by an AR objectgeneration unit (an illustration is omitted) such as an application on aPC, a navigation system, or a game device. An image processing apparatussuch as the head mounted display 100 is connected to an external ARobject generation unit via a wired or wireless communication unit 505.Alternatively, the AR object generation unit may be included within theimage processing apparatus.

The real object detection unit 2402 detects a position, posture, or sizeof a real object (a hand of a user or the like), which is a target ofvisual feedback, for example, by performing image recognition for aphotographic image of the outside camera 504. Alternatively, the realobject detection unit 2402 detects a position, posture, or size of areal object by image recognition, by receiving a photographic image ofthe environment camera 703 via the communication unit 505.Alternatively, the real object detection unit 2402 may receive, via thecommunication unit 505, information of a position, posture, or size of areal object, which is obtained by having an external apparatus (notillustrated) recognize a photographic image of the environment camera703.

Note that, it may not be necessary for the real object detection unit2402 to detect all of the real objects included in a photographic imageof the outside camera 504 or the environment camera 703. For example, adetection target may be limited to a prescribed object attempting toperform an operation to an AR object, such as a hand of a user or apointer which a user is gripping. This is because it may not benecessary for an object not able to perform an operation to an AR objectto perform visual feedback.

A between-objects relationship determination unit 2403 acquires positionrelationship information showing a position relationship between an ARobject and a real object, and generates control information for giving avisual effect corresponding to the position relationship information toan image of the AR object, by comparing a position posture of the ARobject generated by the AR information acquisition unit 2401, and aposition posture of the real object detected by the object detectionunit 2402, and by determining a position relationship between the ARobject and the real object (that is, whether the real object is in frontof the AR object, touching the AR object, or behind the AR object).

Note that, it may not be necessary for the between-objects relationshipdetermination unit 2403 to make all of the real objects in aphotographic range of the outside camera 504 or the environment camera703 a target of determination. For example, a determination process of aposition relationship may be performed, by limiting to a real objectpresent in an effective range of visual feedback such as shown in FIG.23. By limiting an effective range of visual feedback, the drawing loadof an AR object of a later stage can be reduced, and an operation of auser being obstructed due to useless visual feedback can be prevented.

Further, the between-objects relationship determination unit 2403 maydetermine that a real object is touching an AR object, not only in astate where the real object is just touching the AR object, by also in astate within an effective range of operation set to a prescribeddistance close to in front and behind a depth direction from the ARobject (refer to FIG. 14). By setting an effective range of operation bythe real object, the success rate of the operation to the AR object bythe real object can be improved.

An AR image rendering unit 2404 performs rendering for an AR image to bedisplayed in a real space, based on AR information such as a position,posture, size, or texture of an AR object acquired by the AR informationacquisition unit 2401.

In an embodiment, there is a main feature in which the AR imagerendering unit 2404 gives visual feedback, that is, a visual effectexpressing that an AR object has been operated by a real object, to theAR object, in accordance with control information generated based onposition relationship information showing a position relationshipbetween the AR object and the real object by the between-objectsrelationship determination unit 2403. A specific example of visualfeedback can include drawing a shadow of a real object on the surface ofan AR object (for example, refer to FIG. 8 to FIG. 10), displaying aring-shaped indicator of light, which shows that a real object isapproaching an AR object (for example, refer to FIG. 15 to FIG. 17), orthe like. However, the display method of visual feedback is not limitedto these.

An overlapping display unit 2405 displays an AR image drawn by the ARimage rendering unit 2404 overlapping on a real space. For example, inthe case where the image display apparatus is the transmission-type headmounted display 100, the overlapping display unit 2405 displays an ARimage at an appropriate pixel position of a display panel, so that theAR object is observed by a user with the position and posture of the ARobject determined by the AR information acquisition unit 2401. Further,in the case where the image display apparatus is the immersive-type headmounted display 300, the overlapping display unit 2405 overlaps an ARimage at an appropriate location on a surrounding image (a video throughimage) photographed by an outside camera.

FIG. 25 shows a process procedure for performing a drawing process foran AR image attached to visual feedback in the form of a flow chart, inan image processing apparatus including the functional configurationshown in FIG. 24.

First, the AR information acquisition unit 2401 acquires AR informationsuch as a position, posture, or size related to an AR object to bedisplayed overlapping on a real space (step S2501).

Further, the real object detection unit 2402 detects a position,posture, or size of a real object (a hand of a user or the like), whichis a target of visual feedback, based on a result or the like in whichimage recognition is performed for a photographic image of the outsidecamera 504 or the environment camera 703 (step S2502).

Then, the between-objects relationship determination unit 2403 performsa depth determination of the AR object and the real object, based oninformation acquired in the preceding steps S2501 and S2502, anddetermines whether or not the real object is in front of the AR object(in other words, whether the AR object is shielded by the real object)(step S2503).

Here, in the case where it is determined that the real object is infront of the AR object (Yes in step S2503), the between-objectsrelationship determination unit 2403 additionally checks whether thereal object is present within an effective range of visual feedback(refer to FIG. 23) (step S2504).

In the case where the real object is present within an effective rangeof visual feedback (Yes in step S2504), the AR image rendering unit 2404executes a drawing process of visual feedback of the real objectapproaching the AR object (step S2505). The details of the drawingprocess of visual feedback of the real object approaching the AR objectwill be described later.

On the other hand, in the case where the real object is in front of theAR object, and is not present within an effective range of visualfeedback (No in step S2504), it may not able to be said that the realobject is approaching the AR object (or the real object is attempting tooperate the AR object), and so the AR image rendering unit 2404 executesa usual drawing process of the AR object, which does not include visualfeedback (step S2506). However, since the real object is in front of theAR object, a hidden surface process is performed for the region shieldedby the real object.

Further, in the case where the real object is not in front of the ARobject (No in step S2503), the between-objects relationshipdetermination unit 2403 additionally checks whether the real object isbehind the AR object (step S2507).

Here, in the case where the real object is behind the AR object (Yes instep S2507), the AR image rendering unit 2404 executes a drawing processof visual feedback of the real object which is behind the AR object(step S2508). The details of the drawing process of visual feedback ofthe real object which is behind the AR object will be described later.

On the other hand, in the case where the real object is not in front orbehind the AR object (No in step S2507), the AR image rendering unit2404 executes a usual drawing process of the AR object, which does notinclude visual feedback (step S2509). In this case, since the realobject is not overlapping with the AR object (not shielding) in thevisual field of a user, a hidden surface process of the AR object maynot be necessary.

FIG. 26 shows a process procedure for drawing visual feedback of a realobject approaching an AR object in the form of a flow chart, performedin step S2505 of the flow chart shown in FIG. 25.

First, the between-objects relationship determination unit 2403 checkswhether or not the real object is not in contact with the AR object(step S2601).

The between-objects relationship determination unit 2403 determines thata real object is touching the AR object, not only in a state where thereal object is just touching the AR object, but also in a state withinan effective range of operation set to a prescribed distance close to infront and behind the depth direction from the AR object (refer to FIG.14). By setting an effective range of operation by the real object, thesuccess rate of the operation to the AR object by the real object can beimproved.

Here, in the case where the real object is in contact with the AR object(No in step S2601), the AR image rendering unit 2404 generates visualfeedback at the time when the real object is in contact with the ARobject (step S2604).

In the case where a shadow of the real object corresponding to adistance from the AR object is used, as visual feedback, a shadow whichis almost invisible will be generated. Further, an effect such ascausing the portion which the real object is touching to be lit up maybe added to the visual feedback (for example, refer to FIG. 11).

Alternatively, in the case where a ring-shaped indicator of light with asize corresponding to a distance from the AR object is used, as visualfeedback, a ring of light which is small and sharp will be generated, atthe portion which the real object is touching (for example, refer toFIG. 17).

Further, in the case where the real object is not in contact with the ARobject (Yes in step S2601), the between-objects relationshipdetermination unit 2403 additionally checks whether or not the realobject is intersecting with the AR object (step S2602).

In the case where the real object is not intersecting with the AR object(No in step S2602), the AR image rendering unit 2404 generates visualfeedback corresponding to a distance between the AR object and the realobject (step S2605).

For example, a shadow of the real object having a size and sharpnesscorresponding to a distance from the AR object is generated as visualfeedback (for example, refer to FIG. 9 and FIG. 10). Alternatively, aring-shaped indicator of light having a size and sharpness correspondingto a distance from the AR object is generated as visual feedback (forexample, refer to FIG. 15 and FIG. 16).

Further, in the case where the real object is intersecting with the ARobject (Yes in step S2602), the between-objects relationshipdetermination unit 2402 additionally checks whether or not the realobject pushing through the AR object has entered into a warning range inwhich there is the possibility of colliding with an obstacle (anotherreal object) such as a wall behind the AR object (step S2603).

If the real object is outside of the warning range (No in step S2603),the AR image rendering unit 2404 performs a drawing process for the ARobject intersecting with the real object (step S2606).

On the other hand, in the case where the real object enters into thewarning range (Yes in step S2603), the AR image rendering unit 2404causes the display of the AR object 1302 to disappear (or causing it toflash on-off), or generates warning feedback for a user to be able tovisually confirm an object behind the AR object, such as making itsemitransparent.

Then, the AR image rendering unit 2404 performs a drawing process forthe AR object, while performing a hidden surface process for the regionshielded by the real object (step S2608).

To continue, the AR image rendering unit 2404 ends the present processroutine, by performing a drawing process for the visual feedbackgenerated by any of the preceding steps S2604 to S2607, while performinga hidden surface process for the region shielded by the real object(step S2609). In the case where drawing visual feedback on the surfaceof the AR object, the AR image rendering unit 2404 may cause the colorto be enhanced, and may perform visual feedback while retaining thevisibility of the user, by mutually drawing the color of the AR objectand the color of the visual feedback as contrasting colors (refer toFIG. 21 and FIG. 22).

FIG. 27 shows a process procedure for drawing visual feedback of a realobject behind an AR object in the form of a flow chart, performed instep S2505 of the flow chart shown in FIG. 25.

First, the between-objects relationship determination unit 2403 checkswhether or not the real object is present within an effective range ofvisual feedback (refer to FIG. 23) (step S2701).

In the case where the real object is not present within an effectiverange of visual feedback (No in step S2701), it is not assumed that thereal object performs an operation to the AR object, and it may not benecessary to draw this visual feedback, and so the present processroutine ends, by skipping all of the succeeding processes. By limitingsuch an effective range of visual feedback, the drawing load of an ARobject of a later stage can be reduced, and an operation of a user beingobstructed due to useless visual feedback can be prevented.

On the other hand, in the case where the real object is present withinan effective range of visual feedback (Yes in step S2701), the AR imagerendering unit 2404 generates visual feedback corresponding to adistance between the AR object and the real object, such as shown inFIG. 18 to FIG. 20, for example (step S2702).

Then, the AR image rendering unit 2404 ends the present process routine,by performing a drawing process for the AR object (step S2703), andsubsequently, performing a drawing process for the visual feedbackgenerated in step S2702 (step S2703).

In this way, according to the technology disclosed in embodiments of thepresent disclosure, at the time when a user is attempting to perform anoperation by his or her hand or the like to an AR object, under anenvironment which observes a real space in which the AR object isdisplayed, it can be intuitively known how far the AR object isseparated from the user himself or herself, through visual feedback.Further, it becomes easy for a user to touch a position aimed at an ARobject, by making visual feedback a clue.

Further, according to the technology disclosed in embodiments of thepresent disclosure, it becomes easy to also use an AR object as an inputmechanism such as a touch panel, by not only simply displaying an ARobject overlapping on a real space, but also adding UI parts such asmenu buttons.

Further, according to the technology disclosed in embodiments of thepresent disclosure, since space perception becomes easy by adding visualfeedback to an AR object, a user can easily perform an operation such astouching a complex three-dimensional shaped AR object.

INDUSTRIAL APPLICABILITY

Heretofore, the technology disclosed in embodiments of the presentdisclosure has been described in detail, while referring to specificembodiments. However, it is evident that a person skilled in the art canperform corrections or substitutions in a range which does not deviatefrom the content of the technology disclosed in thespecifically-described embodiments of the present disclosure.

The technology disclosed in embodiments of the present disclosure can beapplied to various image display apparatuses, such as a head mounteddisplay, a head-up display, a small-sized information terminal such as asmartphone or a tablet, a navigation system, or a game device, as adevice which presents an AR object. While the present specification andthe attached figures only describe a two-eye type head mounted display,it is needless to say that the technology disclosed in embodiments ofthe present disclosure can similarly be applied to a single-eye typehead mounted display.

While the calculation process for performing visual feedback can beexecuted within a display apparatus main body which displays an ARobject such as a head mounted display, a calculation process forperforming visual feedback can be performed in an external apparatussuch as a server on a network, and only a display output can beperformed in accordance with this calculation result in the displayapparatus.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

Additionally, the present technology may also be configured as below.

(1) An information processing apparatus including:

circuitry configured toacquire information indicating a spatial relationship between a realobject and a virtual object, andinitiate generation of a user feedback based on the acquiredinformation, the user feedback being displayed to be augmented to agenerated image obtained based on capturing by an imaging device, oraugmented to a perceived view of the real world, wherein acharacteristic of the user feedback is changed when the spatialrelationship between the real object and the virtual object changes.

(2) The information processing apparatus according to (1), wherein asize of display of the user feedback is changed as the spatialrelationship between the real object and the virtual object is changed.

(3) The information processing apparatus according to (1) or (2),wherein a size of display of the user feedback is displayed to besmaller as the real object becomes closer to the virtual object.

(4) The information processing apparatus according to any of (1) to (3),wherein a shape of display of the user feedback is changed when thespatial relationship between the real object and the virtual object ischanged.

(5) The information processing apparatus according any of (1) to (4),wherein a visual quality of display of the user feedback is changed asthe spatial relationship between the real object and the virtual objectis changed.

(6) The information processing apparatus according to any of (1) to (5),wherein at least one of a color, a brightness, or transmittance ofdisplay of the user feedback is changed as the spatial relationshipbetween the real object and the virtual object is changed.

(7) The information processing apparatus according to any of (1) to (6),wherein the color of display of the user feedback is displayed to belighter as the real object becomes closer to the virtual object.

(8) The information processing apparatus according to any of (1) to (7),wherein a movement of display of the user feedback is changed as thespatial relationship between the real object and the virtual object ischanged.

(9) The information processing apparatus according to any of (1) to (8),wherein at least a position, a vibration, or a trajectory of display ofthe user feedback is changed as the spatial relationship between thereal object and the virtual object is changed.

(10) The information processing apparatus according to any of (1) to(9), wherein at least a portion of the virtual object is not displayedas the real object approaches the virtual object.

(11) The information processing apparatus according to any of (1) to(10), wherein at least a portion of the virtual object is lighted up asthe real object approaches the virtual object.

(12) The information processing apparatus according to any of (1) to(11), wherein the spatial relationship includes a distance between thereal object and the virtual object.

(13) The information processing apparatus according to any of (1) to(12), wherein the characteristic of the user feedback is changed basedon whether the real object is within a predetermined distance from thevirtual object.

(14) The information processing apparatus according to any of (1) to(13), wherein a characteristic of the user feedback which is changed ina case that the real object is within the predetermined distance fromthe virtual object is different from a characteristic of the userfeedback which is changed in a case that the real object is not withinthe predetermined distance from the virtual object.

(15) The information processing apparatus according to any of (1) to(14), wherein the circuitry is further configured to set thepredetermined distance in front of and behind the virtual object as arange of operation of the virtual object.

(16) The information processing apparatus according to any of (1) to(15), wherein the virtual object overlaps the real object when the realobject is pushed through a location corresponding to a display positionof the virtual object.

(17) The information processing apparatus according to any of (1) to(16), wherein a warning feedback is given when the real object is pushedthrough a location corresponding to a display position of the virtualobject.

(18) The information processing apparatus according to any of (1) to(17), wherein display of at least a portion of the virtual object isterminated when the real object is pushed through a locationcorresponding to a display position of the virtual object.

(19) The information processing apparatus according to any of (1) to(18), wherein the user feedback includes a display portion for an inputoperation performed by a user.

(20) The information processing apparatus according to any of (1) to(19), wherein the spatial relationship includes at least one of adistance, a direction, or posture information between the real objectand virtual object.

(21) The information processing apparatus according to any of (1) to(20), wherein the user feedback includes a sound or a vibration.

(22) The information processing apparatus according to any of (1) to(21), wherein the user feedback is stereoscopically displayed.

(23) The information processing apparatus according to any of (1) to(22), wherein the characteristic of the user feedback is changedstepwise or in relation to a changing of the spatial relationshipbetween the real object and the virtual object.

(24) The information processing apparatus according to any of (1) to(23), wherein a magnitude of the characteristic of the user feedback ischanged in relationship to a changing of the spatial relationshipbetween the real object and the virtual object.

(25) The information processing apparatus according to any of (1) to(24), wherein a location of displaying the user feedback is determinedbased on a location of a target whose position is indicated by atrajectory direction of the real object.

(26) An information processing method including:

acquiring information indicating a spatial relationship between a realobject and a virtual object;generating a user feedback based on the acquired information anddisplaying the user feedback to be augmented to a generated imageobtained based on capturing by an imaging device, or augmented to aperceived view of the real word, wherein a characteristic of the userfeedback is changed when the spatial relationship between the realobject and the virtual object changes.

(27) A non-transitory computer-readable medium having embodied thereon aprogram, which when executed by a computer causes the computer toexecute a method, the method including:

acquiring information indicating a spatial relationship between a realobject and a virtual object;generating a user feedback based on the acquired information; anddisplaying the user feedback to be augmented to a generated imageobtained based on capturing by an imaging device, or augmented to aperceived view of the real word, wherein a characteristic of the userfeedback is changed when the spatial relationship between the realobject and the virtual object changes.

(28) An information processing apparatus including:

an acquisition unit which acquires position relationship informationshowing a position relationship between a virtual object visuallyrecognized by a user at a same time as an image of a real space, and areal object performing an operation instruction to the virtual object;anda generation unit which generates control information for controlling anoutput based on the position relationship information, wherein thegeneration unit,in a case where it is decided the virtual object and the real objecthave a first position relationship based on the position relationshipinformation, generates first control information for performing a firstoutput, andin a case where it is decided the virtual object and the real objecthave a second position relationship different to the first positionrelationship based on the position relationship information, generatessecond control information for performing a second output different tothe first output.

(29) The information processing apparatus according to (28), wherein, ina case where it is decided the virtual object and the real object have athird position relationship different to both of the first and thesecond position relationships based on the position relationshipinformation, the generation unit generates third control information forperforming a third output different to both of the first and the secondoutputs.

(30) The information processing apparatus according to (29), wherein thegeneration unit generates the first, second, and third controlinformation for respectively performing the first, second, and thirdoutputs including visual information.

(31) The information processing apparatus according to (30), wherein thegeneration unit generates the first to third control information forperforming an output which adds a visual effect to an image of thevirtual object in at least one output from among the first to thirdoutputs.

(32) The information processing apparatus according to (28), wherein thegeneration unit generates control information for performing an outputwhich includes information showing a distance between the virtual objectand the real object.

(33) The information processing apparatus according to (28), wherein thegeneration unit generates control information for performing an outputwhich includes information showing a distance in a depth direction seenfrom the user between the virtual object and the real object.

(34) The information processing apparatus according to (31), wherein, ina case where it is decided the real object is in front of the virtualobject, in a case where it is decided the real object is in contact withthe virtual object, or in a case where it is decided the real object isbehind the virtual object, the generation unit generates the controlinformation for performing an output having a different visual effect.

(35) The information processing apparatus according to any of (28) and(29),

wherein, at the time when the real object is present in an effectiverange of operation set within a prescribed distance in front and behindthe virtual object, the generation unit generates control informationfor performing an output showing the real object is in contact with thevirtual object.

(36) The information processing apparatus according to (35), wherein thegeneration unit generates control information for prompting the user tocorrect a position of the real object by notifying the user that thereal object is not in contact with the virtual object while allowing acontinuous operation of the virtual object within the effective range ofoperation.

(37) The information processing apparatus according to any of (28) to(35), wherein, only in a case where the real object is present within aneffective range of a visual effect set for the virtual object, thegeneration unit generates control information for outputting a visualeffect to an image of the virtual object.

(38) The information processing apparatus according to (37), wherein thegeneration unit sets an effective range of the visual effect to a fixedrange which includes the virtual object.

(39) The information processing apparatus according to any of (28) to(38), wherein, in a case where the real object is in front of thevirtual object or is in contact with the virtual object, the generationunit generates control information for outputting a visual effect on thevirtual object.

(40) The information processing apparatus according to (39), wherein thegeneration unit generates control information for outputting, as thevisual effect, a shadow of the real object having a size and sharpnesscorresponding to a distance from the virtual object to the real object.

(41) The information processing apparatus according to (39), wherein thegeneration unit generates control information for outputting, as thevisual effect, a ring-shaped indicator of light having a size andsharpness corresponding to a distance from the virtual object to thereal object.

(42) The information processing apparatus according to (39), wherein, ata time when the real object is intersecting with the virtual object, thegeneration unit generates control information for outputting a warning.

(43) The information processing apparatus according to (42), wherein thegeneration unit outputs a warning sound showing danger of the realobject colliding with another real object present behind the virtualobject, or generates control information for outputting a warningincluding a visual effect where a color of the another real object ischanged.

(44) The information processing apparatus according to (39), wherein thegeneration unit generates control information for outputting an visualeffect for drawing the virtual object by taking into consideration ashielding region by the real object.

(45) The information processing apparatus according to (39), wherein thegeneration unit generates control information for outputting the visualeffect by taking into consideration a shielding region by the realobject.

(46) The information processing apparatus according to any of (28) to(38), wherein, in a case where the real object is behind the virtualobject, the generation unit generates control information for outputtinga visual effect corresponding to a distance from the virtual object tothe real object.

(47) The information processing apparatus according to (28), where thegeneration unit generates control information for outputting the visualeffect with a contrasting color to that of the virtual object.

(48) The information processing apparatus according to (28), wherein thegeneration unit generates control information for outputting a visualeffect which draws a portion to add the visual effect from within thevirtual object with a contrasting color to an original color.

(49) An information processing method including:

acquiring position relationship information showing a positionrelationship between a virtual object visually recognized by a user at asame time as an image of a real space, and a real object performing anoperation instruction to the virtual object; and generating controlinformation for controlling an output based on the position relationshipinformation,wherein, generating control information includes,in a case where it is decided the virtual object and the real objecthave a first position relationship based on the position relationshipinformation, generating first control information for performing a firstoutput, andin a case where it is decided the virtual object and the real objecthave a second position relationship different to the first positionrelationship based on the position relationship information, generatingsecond control information for performing a second output different tothe first output.

(50) An image display apparatus including:

a display unit which displays a virtual object so as to be visuallyrecognized by a user at the same time as an image of a real space,an acquisition unit which acquires position relationship informationshowing a position relationship between the virtual object, and a realobject performing an operation instruction to this virtual object, anda generation unit which generates control information for controlling anoutput of the display unit showing a position relationship between thevirtual object and the real object, based on the position relationshipinformation,wherein the generation unit,in the case where it is decided that the virtual object and the realobject have a first position relationship based on the positionrelationship information, generates first control information for thedisplay unit to perform a first output, andin the case where it is decided that the virtual object and the realobject have a second position relationship different to the firstposition relationship based on the position relationship information,generates second control information for the display unit to perform asecond output different to the first output.

(51) The image display apparatus according to (50), further including avirtual object generation unit which generates the virtual object.

(52) An image display method, including:

acquiring position relationship information showing a positionrelationship between a virtual object visually recognized by a user atthe same time as an image of a real space, and a real object performingan operation instruction to this virtual object,generating control information for controlling an output based on theposition relationship information, anddisplaying the virtual object so as to show a position relationship withthe real object, based on the control information,wherein, generating control information includes,in the case where it is decided that the virtual object and the realobject have a first position relationship based on the positionrelationship information, generating first control information forperforming a first output by the displaying, andin the case where it is decided that the virtual object and the realobject have a second position relationship different to the firstposition relationship based on the position relationship information,generating second control information for performing a second outputdifferent to the first output by the displaying.

(53) A computer program written in a computer-readable format so as tocause a computer to function as:

an acquisition unit which acquires position relationship informationshowing a position relationship between a virtual object visuallyrecognized by a user at a same time as an image of a real space, and areal object performing an operation instruction to the virtual object;anda generation unit which generates control information for controlling anoutput based on the position relationship information,wherein the generation unit,in a case where it is decided the virtual object and the real objecthave a first position relationship based on the position relationshipinformation, generates first control information for performing a firstoutput, andin a case where it is decided the virtual object and the real objecthave a second position relationship different to the first positionrelationship based on the position relationship information, generatessecond control information for performing a second output different tothe first output.

REFERENCE SIGNS LIST

-   100 head mounted display-   101L, 101R virtual image optical unit-   102 supporting body-   103L, 103R microphone-   104L, 104R display panel-   300 head mounted display-   303L, 303R microphone-   304L, 304R display panel-   305 outside camera-   306 interpupillary adjustment mechanism-   501 control unit-   501A ROM-   501B RAM-   502 input operation unit-   503 remote control reception unit-   504 outside camera-   505 communication unit-   506 storage unit-   507 image processing unit-   508 display driving unit-   509 display unit-   510 virtual image optical unit-   511 audio processing unit-   512 audio input and output unit-   703 environment camera-   2401 AR information acquisition unit-   2402 real object detection unit-   2403 between-objects relationship determination unit-   2404 AR image rendering unit-   2405 Overlapping display unit

What is claimed is:
 1. An information processing apparatus including: circuitry configured to acquire information indicating a spatial relationship between a real object and a virtual object, and initiate generation of a user feedback based on the acquired information, the user feedback being displayed to be augmented to a generated image obtained based on capturing by an imaging device, or augmented to a perceived view of the real world, wherein a characteristic of the user feedback is changed when the spatial relationship between the real object and the virtual object changes. 